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(57) ABSTRACT 

A method of anonymous targeted insertion of indicia into 
video broadcasts. Individual televisions or other video 
reception devices are associated with set-top boxes that 
monitor the usage and viewing habits of the television set or 
other video reception device. A viewer profile derived from 
data acquired from said monitoring is created wherein the 
viewer profile indicates certain characteristics about the 
viewer. This profile is transmitted to a centralized database, 
said centralized database being an intermediate link between 
the origin of the video broadcast and the end viewer. The 
purpose of the database is to link specific insertable indicia 
with matching specific viewer profiles. The insertable indi- 
cia are encoded directly into the broadcast video and 
re-broadcast to the end viewer where the set-top box 
decodes the broadcast video and performs insertion of the 
indicia. Thus, the system and method allow advertisers to 
target specific ads or indicia to specific viewing profiles. 

16 Claims, 4 Drawing Sheets 



0^ 





07/09/2004, EAST Version: 1.4.1 



U.S. Patent Sep. 3, 2002 Sheet 1 of 4 US 6,446,261 Bl 



FI6. 



"1 , . 



•10 



ri2 



14 



15 



VIDEO 
PRODUCTION 
UNIT 



18 




7|v-i3 



20 



RECOGNITION 
UNIT 



TRACKING 
UNIT 



22 



16 



24 



OCCLUSION 
MASK UNIT 



CAMERA HEAD 

DATA I- 1 
INTERPRETER 



26 



27 



28 



ENCODING 
UNIT 



44 "\ 



_ _ UPSTREAM LIVE_VI0E0 INSERTION_SYSTEM 

0. !>■ 1 STUDIO SITE^ ^ 



52- 



DECOMPRESSION 


r 

i 
i 




MEANS 


T 





CONTROL 


I 


MEMORY 




VIEWER 


DEVICES 


i 


DEVICE 




PROFILE 



LsE™f_reyicE_^^ 
44 "\, 1 46 



54- 



I 



DECOMPRESSION 


r 




MEANS 







LT7I 



CONTROL 




MEMORY 


1 VIEWER 


DEVICES 




DEVICE 


| PROFILE 



SET-TOP DEVICE L_J»™!LLVB_„_, 




07/09/2004, EAST Version: 1.4.1 



U.S. Patent Sep. 3, 2002 Sheet 2 of 4 US 6,446,261 Bl 



70 

X BROADCAST *\ 
VIDEO J 



72- 



TUNER 



82- 



76 



1 



DECOMPRESS 



clock 



-84 



BASEBAND 
VIDEO 



VERTICAL 
BLANKING 
INTERVAL 
DECODER 



78 

A. 



z 



TYPE OF 
PROGRAM 



REQUIRED 
VIEWER 
USAGE 
PROFILE 



VIEWER 
USAGE 
PROFILE 



re 



FIG. 2 

71 




r-94 

96 

A. 



VIEWER 
REMOTE 
CONTROLLER 



106 

•?~^\ VIEWER'S "\ 

E ) V TELEVISION J 



104- 



CHANNEL 3 
MODULATOR 



86* 



102 



DELAY 
LINE 




Mooa 

INFORMATION 



88 



100 



MUX 



m 



WARP 



OCCLUSION 
MASK 



87 



VIDEO 
INSERTION 



PROFILE 
MATCHER 



90 



98 



44 







TEXT TO 






VIDEO 



.-1 



07/09/2004, EAST Version: 1.4.1 



U.S. Patent Sep. 3, 2002 Sheet 3 of 4 US 6,446,261 Bl 



FIG. 3 



AVERAGE 
VIEWING 
TIME 
(ROLLING 
AVERAGE 
OVER 
LAST , 
30 DAYS) 




WSMF* TIME Of DAY HATCH60) 



T 

122 



FIG. 5 




MOVIES 16-03X1 

SIT. COM 14.7% 

CARTOONS 10.23* 

NEWS 10.04X 

DRAMA 8.4% 

SOAP OPERA 4.57% 

SPORT 4. 1% 

CHILOflEN'S SHOWS 3.7% 

GAME SHOWS 3.56% 

RELIGIOUS SHOWS 3.56% 

CRIME SHOWS 2.74% 

MUSIC SHOWS 2.56% 

TALK SHOWS 2.46% 

INFORMATION SHOW 2.46% 

COMEDY 2.19% 

INFOMERCIALS 1.46% 
ENTERTAINMENT SHOW 1.37% 

ACTION SHOWS 1.1% 

SCIENCE FICTION 0.91% 

SCIENCE 0.82% 

SHOPPING SERVICE 0.64% 

HEALTH SHOW 0.64% 

MYSTERY SHOW 0.553 

WESTERN SHOW 0.181 

EDUCATION 0.09% 

COMEDY VARIETY 0.09% 



07/09/2004, EAST version: 1.4.1 



U.S. Patent Sep. 3, 2002 Sheet 4 of 4 US 6,446,261 Bl 



130 



132 



I ( INTERNET/TELEPHONE ) ^CABlT) 



l: 

r 



I 



MODEM 



•138 



FIG. A 

134 

- -jl — - --- 

fBTO^CgT) C SATELLITE Yf 



70 



136 



I 



CABLE 
MODEM 



n 



■140 



142- 



TUNER 



152 



DATA 
ROUT EH 



TEXT TO 
VIDEO/ 
AUDIO 



166 



DATA 
STORE 




VIEWER 
USAGE 
PROFILE 
STORE 



-170 



interpreter] 

M64 



CENTRAL 
CONTROLLER 



-172 



LOCATION 
INFORMATION 



CLOCK 



174 



3= 



DECODER 



-144 



L 



152 



VIDEO/ 
AUDIO 
ROUTER 



VIOEO/AUDIO 
STORAGE 



154 



-146 



X 



148 



VIEWER 
CONTROL 
INTERFACE 



I 




VIOEO/AUDIO 
DECOMPRESS 



156 



VIDEO/AUDIO 
MIXER 



158 



OCCLUSION MASK 
GENERATOR 



-160 



WARPER 



162 



CHANNEL 3 
MODULATOR 



END USER 
TV SET 



T 

44 



I 



VIEWER'S 
HEMOTE 
CONTROL 



07/09/2004, EAST Version: 1.4.1 



US 6,4 

1 

SET TOP DEVICE FOR TARGETED 
ELECTRONIC INSERTION OF INDICIA 
INTO VIDEO 

CROSS-REFERENCE TO RELATED 
APPLICATIONS 

This patent application is related to and claims the benefit 
of U.S. provisional application serial No. 60/034,517 filed 
on Dec. 20, 1996 entitled "Set Top Device for Targeted 
Electronic Insertion of Indicia into Video". 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

This invention relates to set top video reception devices, 
and particularly, enhancing them to allow insertion of tar- 
geted indicia Into video. 

2. Description of the Related Art 

Electronic devices for inserting images into live video 
signals, such as described in U.S. Pat. No. 5,264,933 by 
Rosser, et al. and U.S. Pat. No. 5,491,517 by Kreitman et al., 
have been developed and used commercially for the purpose 
of inserting advertising and other indicia into video 
sequences, including live broadcasts of sporting events. 
These devices are capable of seamlessly and realistically 
incorporating logos or other indicia into the original video in 
real time, even as the original scene is zoomed, panned, or 
otherwise altered in size or perspective. 

Live video insertion of indicia requires several steps. The 
event video must be recognized, tracked, and adjusted for 
the potential insert perspective and occluding objects prior 
to actual insertion. In the systems discussed in U.S. Pat. No. 
5,264,933 by Rosser, et al. and U.S. Pat. No. 5,491,517 by 
Kreitman et al. it was assumed that the broadcaster would 
perform the complete process, including recognition, 
tracking, creating an occlusion mask, warping inserts to 
correctly match the current image, and correctly mixing the 
original video, warped insert and occlusion mask. 

In U.S. Pat. No. 5,543,856 of Rosser, et al., a Live Video 
insertion System (LVIS) split into two functional parts is 
described, with an upstream, "master" part performing rec- 
ognition and occlusion mask generation, and sending this 
information downstream, along with various control 
parameters, to a less computationally endowed downstream 
"slave" part, capable of warping inserts to correctly match 
the current image, and correctly mixing the original video, 
warped insert and occlusion mask. 

A number of current trends in television, video, and 
computer technology make it feasible and economically 
likely that a "slave" LVIS unit will be included within future 
set-top units. One trend is toward broadcasters sending 
compressed video signals directly to the home. Compression 
is driven by a limited availability of broadcast bandwidth, 
especially satellite based broadcasts. In order to decompress 
the compressed video, users require a set-top device that has 
significant computing power and memory. These set top 
devices are required to run decompression algorithms in real 
time. Memory and computing power could also be utilized 
to make the set-top device act as a downstream "slave" part 
of an LVIS system, 

A second trend is the decreasing price of memory and 
computing power thereby increasing a personal computer's 
ability to process video bandwidths of information. 

A third trend is the movement by telephone companies 
and other wire network providers to higher bandwidth 
networks. There is also the possibility that the World Wide 
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Web, or some similar computer network, could become a 
means for large-scale data exchange or broadcast of high 
quality video information. Compressed video, still necessary 
to traverse networks with limited bandwidths, is decom- 

5 pressed by the personal computer receiving the data. The 
video processing power of the personal computer may be 
sufficient to also be utilized as the downstream "slave" 
section of an LVIS system. 

Yet another trend is towards sending digital television 

10 signals directly to the home. This means that the television 
set itself will be a digital processor, potentially powerful 
enough to be programmed and used for the Image warping 
and the other processes required of a downstream slave unit 
of an LVIS system. 

15 In all these scenarios, the significant point, as far as this 
invention is concerned, is that the set-top device, the last link 
of the video transmission chain, has significant computing 
power and memory. When this computing power and 
memory is sufficient for the viewer's set-top device to act as 

20 the downstream or "slave" section of an LVIS system, a very 
interesting possibility arises-the possibility to target adver- 
tising within a mass medium. In particular, it makes narrow 
casting of advertising possible, particularly insertions, in 
television and other video transmissions. 

25 

To understand the benefits of narrow casting to television 
and video audiences, which is the subject of this application, 
it is useful to understand the concepts of targeting advertis- 
ing. 

30 The most pervasive, and precise, of existing methods of 
narrow-casting or target advertising is direct mail (aka 
junk-mail) which uses mail to deliver material to selected 
audiences. The starting point for direct mail is a database of 
addresses. These databases can also be cross-finked to so 

35 called profile factors, or personal information, pertaining to 
the residents at each address. These profile factors are 
typically age, income, family composition, number of chil- 
dren and their ages, type of automobile owned, dwelling 
type, zip code and various other demographic, psycho- 

40 graphic and life-style information. The more of these profile 
factors the data base contains, the more useful it is for 
targeting the advertising. The data base is sorted by com- 
puter to generate a mailing list of candidates whose profile 
factors match an advertiser selected sub-set. The advertiser 

4 5 believes that clients whose profile factors fall within this 
selected sub set will be more responsive to buying the 
product the advertiser is selling, so that by mailing only to 
those people, the advertiser (or their client) can reach all of 
the audience who are highly predisposed to purchase their 

50 product, with the minimum of expense. 

The use of these databases has three problems. The first is 
that they are only effective for mail. The more influential 
mass media, especially television, cannot be targeted, with 
anything like the same geographic precision because of the 

55 broadcast nature of the transmission. The second is the 
problem of trying to keep the data bases up to date. Typical 
sources of data used to compile such data bases, such as 
census information, professional licensed databases, credit 
card transactions, warranty cards, reverse directories and 

60 consumer surveys can be months, and more typically, years 
out of date, leading to considerable waste and to missing a 
substantial fraction of potential prospects. Even good data 
bases only guarantee 80-90% deliverability— i.e. 10-20% 
of the addresses are no longer valid. The third is the concern 

65 for privacy. The existence of such centralized data bases 
worries many people because of their potential misuse by 
agencies, including but not limited to government agencies, 
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having authorized or unauthorized access to the data bases, monitored by a central system. By choosing the sample 

and also their potential use by criminals for targeting theft, households scientifically so that each household in the 

con schemes and other misdeeds. television viewing population has a known chance of 

The set-top downstream version of LV1S solves all three ^ ° btaine ? ^ «"». be 

of these problems. First, it brings the power of direct 5 P* 0 ^ 6 ' 1 to * la 5f televisi0 ° vl6Wln B a ^ 6I1< f • 

, . ^ . . , . t # The sample size required for survey depends on the reh- 

marketing to video, in particular to the mass market medium ^ A k ^ ^ for most 

of television. Moreover it can do this in a way that avoids needs For e le> national pollSf such as those mjsductod 

the need for centralized data bases, with their privacy and by ^ weU ^ QWn Gallup or Harris organizationSj generally 

out-of-date concerns. The proposed targeting mechanism of use samp i cs 0 f a bout 1,500 persons to reflect national 

this, application, Anonymous Target Profiling, effectively 10 atlitudes ^ opinions to within an accuracy of ±4%. A 

targets viewers profile factors without making them publicly sample of this size produces accurate estimates even for a 

available in a way that ensures profile factors are close to country as large as the United-States with a population of 

100% current. over 250 million people. 

In one usage of the invention, a broadcaster would estab- 
lish a continuous survey of a few thousand households of 
The invention comprises both a method and an apparatus known profile factors for each significant broadcast region, 
to act as a Live Video Insertion System (LVIS), split into two These surveys would be used to generate cross-correlations 
functional parts, with an upstream, "master" part doing the between viewer usage profiles and viewer profile factors, 
recognition and occlusion mask generation, and sending this ^ Advertisers wishing to have their advertising targeted to 
information downstream, along with various control viewers with a particular sub-set of profile factors would be 
parameters, to a less computationally endowed downstream able to use the cross-correlations to translate their viewer 
"slave" part, capable of warping inserts to correctly match profile requests into a viewer usage profiles request. The 
the current image, and correctly mixing the original video, broadcaster would then send the required viewer usage 
warped insert and occlusion mask, where the downstream profiles as part of the broadcast in for instance,.the vertical 
section is part of a set-top device in a viewer's home. blanking interval (VBI) along with the advertisers insertion 
Because of the location of the set-top device at the also in the VBI, over a number of fields, if necessary. At the 
viewer's television set, it becomes possible to narrow-cast viewer's set-top, the device would see which insertion was 
video insertions to a single household, which may be a linked to the local viewer usage profile, and insert appro- 
single person or even a particular TV set within a household. 30 pnately. 

Narrow casting could be implemented as the television or For instance, on a widely watched event, such as the super 

video equivalent of direct mailing, in which a central, bowl, a car company may chose to present different models, 

computer sorted data base is used to select viewers whose depending on the demographic or psychographic profile of 

profile factors match an advertiser selected sub-set. For the family, based on their viewing habits. As a simple 

instance, the geographic location of set-top devices could be 35 example, a family with a viewing profile that includes 

made extremely local by GPS type devices in the set-top significant viewing of young children's programs is assumed 

device which may also double as theft protection to have children and may be shown advertisements for a 

mechanisms, or by phone numbers of attached modems, or mini-van, while a family with a profile that includes signifi- 

postal codes, or by mailing addresses which are stored in the cant viewing of programs for out-door sports may be shown 

set top device, possibly as part of product warranty submis- 40 an advertisement for a sports utility vehicle made by the 

sions. However, the availability of significant memory and same company. 

computing power in the set-top device opens up a much There could also be a "write-in" dimension to the viewing 

more exciting possibility, which we term Anonymous Target providing the viewer the opportunity to select extra specific 

Profiling (ATP). profile factors. For instance, viewers who are looking for a 

Anonymous Target Profiling does not require a central- 45 car ma y add ^is fact to their viewer profile in order to 

ized database of all potential clients. Instead, there is a deliberately solicit advertisements for cars. It may also be 

viewer usage recorder or monitor, located at the viewer possible to specify price ranges and other relevant param- 

location, and a viewer usage interpreter or key, supplied with eters. 

the broadcast. The viewer usage recorder or monitor is a In a further embodiment of the invention, insertions could 

system which monitors television usage patterns and stores 50 appear as border advertisements surrounding, or partly sur- 

a continuously updated version of a usage profile. The rounding one or more of the video windows, or could be a 

set-top device is an ideal place to locate a viewer usage separate video window, which may change position, size, 

monitor. In a simple form, the viewer usage monitor would shape and orientation on the screen as a means of increasing 

classify programs (or channels) and record a rolling viewing the impact of the advertisement. 

profile of viewing habits, including type of program 55 In another further embodiment of the invention, the 

watched, time of day and day of the week of viewing the set-top device could be used not only for in-programming 

program and duration of that viewing. More complex mod- advertising, as made possible by the LVTS system, but could 

els of viewer usage may also include programs not watched, use one or more secondary, possibly compressed, video 

intensity of viewing (i.e. volume adjustments), surfing pat- channels as a source of alternate advertisements for showing 

terns (i.e. what video snippets arrest the attention of a $ 0 in the conventional advertising breaks. These inter-program 

channel surfer, even for a short time) and other more subtle advertisements, usually coupled to events in the program, 

aspects of viewer interaction with the medium. could also be at suitable breaks coupled to viewer action, 

A viewer usage interpreter is a key that translates the such as when a viewer first turns the set on, switches 

viewer usage profile into a set of profile factors associated channels, turns the set off, or alters viewing parameters, such 

with the viewing pattern. The viewer usage interpreter could 65 as volume. 

be generated statistically by having a sample of households In a still further embodiment, in which the television is 

of known profile factors, who have their viewing habits also connected to a computer network, such as the World 
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Wide Web, the viewing profile could be extended to include 
a browsing profile, related to frequently visited web sites or 
other services requested. In addition, the advertising inserted 
could be web site addresses or other forms of links to further 
information, or further advertising, related to the product 
being advertised. 

Much of the technology needed to implement the viewer 
usage monitor, necessary for Anonymous Target Profiling, 
could also be used to provide "smart" TV sets, which would 
favorably impact the economics of implementing the inven- 10 
tion by allowing the set-top manufacturers or distributors to 
offset a substantial part of the cost of the set-top device to the 
end user. For example, a smart TV set, when turned on, 
would not just be on the channel it was on when it was 
turned off. Depending on the time and day of the week, it 15 
would turn on to the channel indicated most likely by the 
viewer usage profile, regardless of where it was when it was 
turned off. A smart television may also be used to provide 
user customized burn-ins, especially ones similar to those 
used by broadcasters to show baseball and football statistics. 2 o 
The extra channels and turners necessary for the network to 
offer alternate, full video to its advertisers, could also be 
used to have multiple windows, i.e., enhanced picture in a 
picture. Multiple windows would also enable the ability to 
turn on with predetermined setups more compelling. The 25 
warping necessary for the downstream, slave LV1S system, 
could be used to make one or more of these windows 
re-sizable, magnify able (for people who wanted to examine 
some detail of the video) and even rotatable (for people who 
may want to lie down and have the video on its side as well). 30 
Writable digital video disks, or other high capacity, random 
access memory could be used by advertisers to store full 
motion video for insertion at the appropriate time. Such 
devices can also provide viewers with their own instant 
replay feature, automatically storing the last five or more 35 
minutes of what ever program was being watched. This 
feature would also make the magnification capability more 
compelling, especially for example to sports fans who may . 
wish to go back and look at some aspect of play such as a 
ball landing close to a line in detail. Writable devices can ^ 
also act as a scrap pad for grabbing bits of video they want 
to see later or show someone else; or as a more conventional 
video recorder. 

These additional features may also be used as triggers for 
showing live or still video advertisements, either before or 45 
after the feature is used, or as a border advertisement during 
the use of the feature, or as a live video insertion on some 
recognized part of the video. 

In addition to processing insertion information from a 
pattern recognition version of an LVIS system, the so 
downstream, set-top part of the LVIS system could be taking 
insertion information from camera head sensors, including 
the types of systems developed for virtual studio systems, or 
it could be taking information from multi-user game appli- 
cations. 55 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a schematic diagram showing an LVIS system 
split into an upstream "master" and downstream "slave 
system". 6Q 

FIG. 2 is a schematic diagram, showing details of the 
end-user set-top device, enhanced to enable it to perform as 
the downstream, slave part of an LVIS system. 

FIG. 3 shows an example of a viewer usage profile, shown 
as a bar chart. 65 

FIG. 4 is a schematic diagram, showing details of an 
alternative embodiment the end-user set-top device, 



enhanced to enable it to perform as the downstream, slave 
part of an LVIS system and to act as a smart TV. 

FIG. 5 is a table showing types of television programming 
and the percentage of total air-time within each type in one 
US location in a given week in 1986. 

DETAILED DESCRIPTION OF THE 
PREFERRED EMBODIMENT 

During the course of this description like numbers will be 
used to identify like elements according to the different 
figures which illustrate the invention. 

In the preferred embodiment of the present invention, a 
video transmission, which may, for example, be a live 
television broadcast of an event being played on a court 10, 
is captured for remote viewing by television cameras 12, and 
is composed into a program for viewing within a standard 
video production unit 14, which may be a television pro- 
duction truck or a video studio, equipped with well known 
video production equipment. After being composed into a 
program, the video is fed through the front end of a Live 
Video Insertion System (LVIS) 16. This front end of LVIS 
16 performs the initial functions of recognition using the 
recognition unit 18, tracking using the tracking unit 20 and 
occlusion mask production using the occlusion mask pro- 
duction unit 22, as discussed in detail in U.S. Pat. Nos. 
5,264,933 and 5,543,856, as well as. co-pending patent 
applications: Ser. No. 08/563,598 filed Nov. 28, 1995 
entitled "SYSTEM AND METHOD FOR INSERTING 
STATIC AND DYNAMIC IMAGES INTO A UVE VIDEO 
BROADCAST"; Ser. No. 08/580,892 filed Dec. 29, 1995 
entitled "METHOD OF TRACKING SCENE MOTION 
FOR LIVE VIDEO INSERTION SYSTEMS"; Ser. No. 
08/662,089 filed Jun. 12, 1996 entitled "SYSTEM AND 
METHOD OF REAL-TIME INSERTIONS INTO VIDEO 
USING ADAPTIVE OCCLUSION WITH A SYNTHETIC 
COMMON REFERENCE IMAGE"; and Ser. No. 60/031, 
883 filed Nov. 27, 1996 entitled "CAMERA TRACKING 
USING PERSISTANT, SELECTED, IMAGE TEXTURE 
TEMPLATES", the teachings of which are hereby included 
by reference. 

The recognition and tracking parameters may also be 
provided by sensors 13 attached to the camera itself, and 
interpreted by a camera head data interpreter 15, as used by 
some virtual reality studio systems, and as discussed in 
detail in U.S. Provisional Application Serial No. 60/038,143 
filed on Nov. 27, 1996 entitled "IMAGE INSERTION IN 
VIDEO STREAMS USING A COMBINATION OF 
PHYSICAL SENSORS AND PATTERN RECOGNITION"; 
the teachings of which are hereby included by reference. 

However, unlike the systems discussed in U.S. Pat. No. 
5,264,933, the front end LVIS system 16 does not use this 
information to merge the insertion with the live video. 
Instead an encoding unit 24 inserts the information obtained 
by the other parts of the LVIS front end 16 into the vertical 
blanking Interval of the video or other appropriate cosignal 
such as, but not limited to, a spare audio channel. In 
addition, encoding unit 24 may also insert all or any of a 
graphic or video for insertion 26, a program category code 
27 or one or more user profile and enabling keys 28. This 
may be done over any number of video fields: a user 
enabling key (as discussed in detail of U.S. Pat. No. 5,543, 
856) and one or more viewer usage profile keys 120. The 
output of encoder unit 24 may be a standard video signal 
which may be the well known standard NTSC or PAL 
television signals with extra information encoded in, for 
instance, the vertical blanking interval or an otherwise 
unused audio channel, or it may be such a compressed video 
signal. 
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The signal produced by the LVIS front-end 16 is sent via 
appropriate means 30, which may be a satellite uplink, or 
telephone company lines, etc., to a central studio site 34 for 
possible further processing before being rebroadcast to a 
wider audience, which may be the general public. The 
central facility 34 may be responsible for inserting any or all 
of a graphic or video for later insertion by the downstream 
part of the LVIS system 46, a user enabling key, one or more 
viewer usage profile keys, and a program category code, all 
for use by the downstream part of the LVIS system 46. The 
central studio site 34 would also be responsible for supply- 
ing conventional video advertising which may also be 
targeted using the Anonymous Profile Targeting methodol- 
ogy of this application. After appropriate alterations are 
made by the central studio site 34, the signal is distributed 
via suitable distribution means 40 and 42 which may be a 
satellite transmission system, a cable network, a terrestrial 
broadcast system, computer network or other appropriate 
means of transferring video or television signals to the end 
user, 

The end user has an appropriate reception device 42, 
which may be a cable connection, a conventional TV aerial, 
a satellite dish, a telephone company line or other appro- 
priate means of receiving television or video signals. After 
reception, the signal is fed to a set-top device 44 before 
reaching the end user's video display screen 56 which may 
be a television screen, a computer monitor or other appro- 
priate display medium. The set-top user may have appro- 
priate means for decompressing 52 signals as well as other 
suitable control devices 54 which may perform various 
functions that make the set-top device desirable to the end 
user, such as but not limited to, customized bum-ins, auto- 
matic channel selection on power up and magnification of or 
re-sizing of extra viewing windows. The set- top device 44 of 
the preferred embodiment has, as a minimum, the compo- 
nents of a downstream LVIS system 46, with the ability to 
strip -off, interpret and use the information mixed in with the 
video signal by the up-stream LVIS system 16. In particular, 
the down stream unit 46 is able to use the information 
generated by the recognition unit 18, the tracking unit 20, 
and the occlusion mask production unit 22 to perform 
seamless insertion of still, animated, and live video indicia 
into the video stream in a way that can make the inserted 
indicia appear to the end user as if it were part of the original 
scene 10. 

The set-top device 44 of the preferred embodiment is also 
capable of stripping off, interpreting and using any of a 
graphic or video, a user enabling key, one or more viewer 
usage profile keys 120, and a program category code, each 
of which may have been attached to the video stream by the 
encoding unit 24 or by central studio site facility 34. In 
particular, by comparing the viewer usage profile keys 120 
with the local viewer usage profile 50, different insertions 58 
and 60 may be made on different end users video viewing 
devices 56. The different insertions may be permanently 
stored locally in memory device. 55, or downloaded, there 
during or prior to transmission of the live video transmission 
in which they are inserted. 

The end-user set-top 44 of the preferred embodiment is 
shown in greater detail in the schematic drawing of FIG. 2. 
The input data stream 70, which may be broadcast video or 
another suitable means of transmitting video to an end user, 
including but not limited to analogue or digital television 
broadcast, or MPEG2 or other compressed video, is typi- 
cally received via a selection device 72, which may be, but 
is not limited to, a standard television tuner. The function of 
the selection device 72 is to discriminate between the variety 
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of different video programs or data streams which may be 
being distributed over the same channel but on different 
frequency bands, or deriving from different locations on a 
network. In the preferred embodiment the selection device 

5 72 is monitored by a viewer usage profile generator 74, 
which also has access to the current time and date via a clock 
76 and to the type of program from the type of program 
indicator 78, which has been given this information by the 
vertical blanking interval decoder 80. The function of the of 

10 the usage profile generator 74 is to build up a history of the 
use of video viewing. The pattern of viewing, which may 
include type of program, nature of program being watched, 
time of day watching the program, day of week of watching 
the program, duration of watching of the program and any 

15 other relevant information pertaining to the program, can be 
used to predict, within acceptable margins of error, the so 
called profile factors which direct market advertisers cur- 
rently obtain from demographic and psycho-graphic data 
bases. 

20 The profile factors could be determined to different 
degrees of accuracy from different amounts of data. For 
instance, a viewing profile which only took into account 
time of day, day of week, time of viewing, duration of 
viewing, including channel surfing patterns — i.e. all data 

25 that requires no program or channel labeling — would reveal 
a great deal about the viewer but would leave substantial 
room for error. For instance, a pattern that had viewing on 
weekdays from 6:30 AM to 8:00 AM but nothing again until 
7:00 PM to 11:00 PM, and nothing on Saturdays until after 

30 10:00 AM, would indicate a house hold with no children, all 
members of which work. While this is a useful conclusion, 
and may be used for a degree of targeting, it does not 
necessarily give any indication of, for instance, the actual 
size of the viewing family, or the gender make-up of the 

35 family. By including monitoring of which program types 
were viewed, considerably more analysis is possible, with a 
good probability of being able to predict such important 
profile factors as gender, age and income. 

In the preferred embodiment, the interpretation of the 

aq viewer usage profiles i.e. the cross — correlation between 
viewer usage profile and viewer profile factors which we 
have termed the viewer usage profile key would be estab- 
lished using well known survey sampling techniques, and 
practiced by such companies as the well known Gallup or 

45 Harris organizations. The viewer usage profile key could be 
generated by having a sample of house holds, of known 
profile factors, who have their viewing habits automatically 
monitored by a central system, which may be a computer 
linked into the viewers set top device by a modem and 

50 telephone link, or other appropriate technology. By choosing 
the sample households scientifically so that each household 
in the television or video viewing population has a known 
chance of selection, the results obtained from the sampling 
can be reliably projected the television or video viewing 

55 public. 

The accuracy and significance of the viewer usage profile 
key will depend on the data used, and how it is used, to 
obtain viewer usage profile 120. A diagrammatic view of a 
viewer usage profile is shown as a bar chart in FIG. 3. 

60 Horizontal axis 122 is used to represent program category, 
and vertical axis 126 is used to represent a measure of 
viewing intensity associated with each program category. A 
simple form of that measure is duration of viewing of a 
program category. A typical entry can be represented as a bar 

65 124. In a simple embodiment, the program categories would 
be time of day and day of week. For example, the first 
category (1, 1) may be a program shown between 12:00 
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midnight and 1:00 AM on Monday. The total number of reduces the weighting of previous weeks. One simple way to 

categories in such a scheme could be 168, i.e., the number do this is to add the previous average of each program 

of hours in a week. Vertical axis 126 in this scheme would category to the current weekly total for each program 

represent time spent viewing any particular category, shown category and divide by two. Many other algorithms could be 

graphically by the height of corresponding bar 124. In $ devised to achieve a similar result. 

general, the program category would be a vector string, each [ n the preferred embodiment, after leaving tuner 72, the 
element of the string corresponding to an attribute. signal goes to the de-compressor, which, if necessary does 
Attributes could include time of day, day of week, month of any decompressing such as, but not limited to, well known 
year, day of year, type of program, channel of viewing, MPEG2 decompression. The output is a base-band video 
broadcaster of the video being viewed, sponsors of material 10 signal 84, which is split into two, one copy of the signal 
being viewed, whether the program being watched was a going to a delay line 86, and the other part to the vertical 
rerun, when the program was made, where the program was blanking interval decoder 80. The function of vertical blank- 
made, producer of the program, major actors in the program, ing interval decoder 80 is to extract the information that was 
director of the program, or any other relevant attribute. The placed there upstream by either LVIS front-end 16 or by 
program category vector generally consists of three 15 central studio site 34. In particular, vertical blanking interval 
attributes the day of week, time of day, and program type. decoder 80 extracts model information 88, occlusion mask 
This necessitates a coded form of the type of program 87, the images or videos to be inserted 90, any auxiliary text 
being transmitted by the broadcasters, preferably in the information 92 associated with the insertion, the required 
vertical blanking interval, though it may be. on or encoded viewer profiles 94 associated with the different insertion 
in a spare audio channel, or in the video itself, either in some 2 o v ^eos 90, and different texts 92. 

spare or extra fields in for instance, the title, or opening For instance, one use may be to have a single video 

sequence, or in the credits. The program type may be insertion 90 of a product, but with a number of different texts 

encrypted in the video itself either as a burn in or some 92. The default text may be in English, but for viewer usage 

alteration to a bum in. In a digital broadcast it may be profiles 74 that show usage of particular ethnic channels, 

encrypted as a least significant bit pattern, as is becoming 25 such as Spanish language channels, the text may be in 

common in digital image authentication schemes. Typical Spanish. Matching a viewer usage profile 74 of the current 

program types might include specific sports, (football, set-top device 44 and the required viewer usage profile 94 is 

baseball, basketball, etc.) each of which may have sub- done by profile matcher 96 which selects required text data 

categories, such as major league, minor league, news, cur- 92 to be fed to text- to-video converters 98. Profile matcher 

rent affairs, film (with sub-categories). 30 96 also selects which of the stored video insertions 90 are 

Richard F. Taflinger, Professor at the Edward R. Murrow fed to warp unit 100. Warp unit 100 takes the appropriate 

School of Communication, Washington State University in model information 88 and uses it to warp the appropriate text 

"Sitcom: What It Is, How It Works", lists twenty-six differ- video 98 and the appropriate video insertion 90 into the 

ent primary types of television shows. FIG. 5 shows a list of appropriate pose required to make the insertion behave as if 

both these twenty-six types and the percentage of each 35 it were part of the natural scene. 

which was available in a given week's television transmis- Occlusion mask 87 is also fed to a warper 89 which uses 

sion in 1986 from an area covered by seven program model information to warp the occlusion mask into the 

providers. Professor Taflinger also notes that the average appropriate pose for the final video. Mixing unit 102 then 

family in 1992 watched television seven hours and seven- combines the warped occlusion mask, the warped insertion 

teen minutes a day, or over 50 hours a week, more than the 40 video and text-video with base-band video 84 which has 

average work week. Television is obviously a major com- been delayed by delay line 86 for the time taken to decode 

ponent of American life, and because of both the diversity of and warp the images into place. The composite output of 

viewing available, and the time spent viewing, patterns of mixing device 102, which is a video signal with an insertion 

TV viewing can be a very powerful tool for determining in place is fed into a channel modulator 104, which converts 

both demographic and psychographic make up of the view- 45 the base-band video to the form expected by the selected 

ing family. Program types could be sub -divided. One major channel of a standard NTSC television set, as is customary, 

subdivision could be whether a program was a re -run or not, Obviously similar arrangements could be made for other 

and if a re-run, how recent. television formats, such as but not limited to the well known 

Vertical axis 126 of the example viewer usage profile 120 PAL, SECAM, digital and HDTV, and other channels of the 

would record the viewing intensity experience associated so television receiver. The resultant signal is then sent to the 

with each of the program categories represented by hori- end user's television set 106 for viewing by the end user, 

zontal axis 122. A simple measure of viewing intensity is An alternative, more generalized version of the set top 

time spent viewing that particular program category. This device is shown schematically in FIG. 4. The input data 

may be represented as total accumulated time, the total time stream 70 can now be one of a number of communication 

in the last month, or some rolling time average. Additional 55 channels, including but not limited to, a telephone/internet 

factors, such as but not limited to, volume, increases in connection 130, a cable video connection 132, a broadcast 

volume, whether or not the set was already tuned to the video aerial 134 and a satellite dish 136. Each of these data 

channel, whether the viewer joined the program late, or streams is selected by the appropriate selection device, 

whether the viewer left the program early, could be used including but not limited to, a modem 138, a cable modem 

weight the time watched to more satisfactorily generate a 60 140, a television tuner 142 and a satellite decoder 144. Each 

compound merit function or estimate of the intensity of the of the selection devices is controlled via a central controller 

viewing experience associated with each of the program 146, which may be a programmable microprocessor, which 

categories. In the preferred embodiment, viewing intensity is in turn controlled by the user, by some device such as a 

associated with each of the program categories is simply the conventional television remote control 71, or some modified 

un-weighted, rolling average of time per week spent watch- 65 version thereof, via the viewer control interface 148. From 

ing that category, with a weighting function that gives the the selection devices, the incoming signal goes either to a 

current week unit weighting, and then systematically video and audio router 150 or a data router 152, both of 



07/09/2004, EAST Version: 1.4.1 



US 6,446,261 Bl 
11 12 

which are under the control of central controller 146. The control interface 148, and information gleaned from inter- 
video and audio router 150 is linked to a video and audio preter 164, to build up a viewer usage profile 120 and store 
storage device 152, which may be a well known electronic this in the viewer usage profile store 170. Viewer usage 
RAM memory or a well known device, such as but not profile 120 stored in viewer usage profile store 170 is a 
limited to a writable Digital Video Disk (DVD), and a 5 measure of the temporal pattern and viewing intensity 
number of video and/or audio processing devices perform- associated with each of the program categories available to 
ing well known functions or operations, including, but not me v i ewer an d discernible by central controller 146. Typi- 
limited to, a decompression device 154, a video and audio cally the ^gram categories would consist of a list or vector 
mixing device 156, an occlusion mask generator 158, an of attributes, where the attributes may include, but are not 
video or image warper 160, and a channel modulator 162. 10 limited to time of day, day of week, month of year, day of 
Each of these devices is under the control of the central year, type of program, channel of viewing, broadcaster of the 
controller, as is the video storage device 152. The video video bemg v i eW ed, sponsors of material being viewed, 
router 150 is also linked to a video and audio interpreter 164, whether the program being watched was a rerun, when the 
which is capable of extracting data embedded in, or attached program was made, where the program was made, producer 
to the video or audio channels, as for instance data included 15 of me pr0 gram, major actors in the program, director of the 
in the vertical blanking interval. The data router 152 has program, or any other relevant information, 
connections to a text-to-video and/or audio converter 166, Eacfa ' of these attributes may itself have a number of 
which can be fed on to the video router 150, the interpreter and ^.^j^ For instancc , lhc important 

164, and a data storage device 168, which is also linked to of of television or vhAso ro ram may 

the Interpreter 164 and is ; under the control of the central M ^ fa ^ ]ia ^ d such as moy{ ^ situation 

controller 146. The central controller also has connection to com ^ m ^ ^ 

the viewer usage profile store 170 and to a location infer- sho ^ &ho ^ rcligkms ^ Cfime sk)WS> 

mation store 172 and a clock 174, which is capable of musjc shows> ^ shows> information shows? 
supplying time and date information. The location informa- mfomercials , entertainment shows, action shows, science 
tion store may include a well known Global Position Sat- M ^ shopping services? health showSj mystery shows> 
ellite (GPS) sensing and interpreting device. wstom sh()WS ^ education shows . Sport may be subdi- 

In normal operation the viewer interacts with their tele- yided for instancej Dy broad categories such as five or 
vision set via the remote control device 71, or other similar rec0 rded, by continent or country of origin, or by type of 
viewer controlled device such as but not limited to, buttons sport sllch as ^ but not ii m j ted to baseball, football, hockey, 
or switches on the viewer set or set top device. The viewer 30 basketballf tennis, soccer, rugby, cricket, bowling, track and 
operations of turning the set on or off; channel selection; fieId etc £ ven me t ype of sport may be subdivided by such 
adjustment of parameters including, but not umited to, cate gories as amateur or professional, international or local, 
volume, brightness, contrast etc., and other viewer usage prc . S eason, regular season, or post-season, major league or 
choices are handled by the viewer control interface 148, minor league 

which may be a graphic user interface displayed on the 35 sim ^ dy most of the types of television or video pro- 
viewers television or video disptay. The viewer requests are » subdivided by broad or narrow classes along 
passed on to the central controller 146 which is typically a ^ iate tQ ^ ^ of ^ thoug £ 
programmed micro-processor, as is well known in the art of ^ ^ * ^ ^ autQmated ^ recognition t0 
embedded control technology. recognize some of these types of programs it is assumed that 

One of the functions of the central controller is to carry 40 mQSt of ^ ^fon^on regarding program type will be 
out the viewer instructions by setting up the appropriate encrypted ia or atta ched to the broadcast or transmission of 
connections between all the appropriate modules within the ^ vMeo Qf television sigasl upstr eam of the set top device, 
set top device. This includes selection of the modem, cable u L . • 1 ^ * a a- a a e 

modem, tuner or decoder as the primary receiving device; ^ °? e met f tod ™** be to include * c ° d «= fo 

setting up of that primary receiving device to the appropriate 45 * e ^ e of *°V™. ™ appropnate part of the vertical 
channel, bandwidth oraddress to .tceive the data or program b !^S ?*™ l > « f » £ ded 0 » staU ° n 

requested by the user, and using the data and video routers identification burn in. Interpreter 164 may stnp the appro- 

- . A 4 . \ , . . . , . . . • , • pnate program type information out 01 the video or televi- 

to direct the television video, audio and data signals via the v\* / f f ** . L .... 

appropriate storage and processing devices, including but sion signal send it to the central controUer for use in building 

not Umited to the video and audio storage unit 152, the so U P local ™ msa S e P rofile - 

decompression unit 154, the video and audio mixer 156 the Another method may be to make use of the program codes 
occlusion mask generator 158, the warper 160 and the supplied by many broadcasters that allow easy programming 
channel modulator 162, so that the viewer ends up with the of television sets, or simply to provide the information about 
information requested, which may be a television program, what settings get what channel to the central controller, 
or a text or image page in hyper text mark up language 55 In addition, in a statistically representative sample of the 
(HTML) or virtual reality modeling language (VRML) or viewing population, central controller 146 would send the 
other suitable protocol, from the world wide web, or some same information being used to compile the local viewer 
combination of such sources, displayed in the appropriate usage profile 170 back via the data router and one or other 
form on their end viewing device 106, which may be a of the internet/telephone modem or the cable modem, back 
television set or a computer monitor or other suitable means 60 to a central collection point. At that central site, the data 
of displaying video or television information. would be correlated and compared with databases contain- 
In addition, the central controller 146 is monitoring the ing demographic and psychographic information about the 
viewer's choices by monitoring the download or interpre- same statistical sample of the viewing population. Signifi- 
tation devices whether they be a modem 138, a cable modem cant correlations between the viewer usage profiles and 
140, a television turner 142 or a satellite decoder 144 and 65 important profile factors, such as but not limited to, age, 
using the settings or mode of those devices, along with data gender, income, ethnic origin, life style (e.g. married, 
from the clock 174, location information unit 172, viewer single), taste, spending habits, credit card usage, employ- 
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ment status, risk taking profile, education etc. could be 
extracted. These correlations would provide a key for adver- 
tisers wishing to target families or individuals with particu- 
lar profile factors. A significant advantage of using these 
correlations would be on popular shows, where a large 
cross-section of the population was viewing. The viewing 
audience could be segmented by profile factors allowing the 
broadcaster to sell different segments of the audience to 
different advertisers, even withia regions covered by a single 
broadcast transmission device. This capability will be par- 
ticularly useful to satellite providers where the transmission 
typically has a large geographical footprint. 

One method of achieving this market segmentation by 
profile factor, is the following. While set-top central con- 
troller 146 is routing the viewer requested video, television 
or other source to the end user's set, it may also be routing 
alternate video or television feeds, either by different chan- 
nels in the same down loader, or by different down-loader to 
video and audio storage unit 152 or data store 165. This 
alternate video feed would typically be relaying a number of 
different advertisements with a requested viewer usage 
profile or range of profiles suitably associated, attached to or 
encoded in, each particular advertising sequence. The con- 
tents of the alternative feed may be stored in video and audio 
storage unit 152. At the appropriate time and place for 
advertising insertion, which may be, but is not limited to, a 
conventional advertising break, or when the viewer changes 
channel, or when a particular image or scene is in view, the 
central controller will use video and audio router 150 and 
data router 152 and whichever is necessary of the other 
video and audio function modules, including but not limited 
to, video and audio storage device 152, to place an appro- 
priate advertisement on the end user's viewing device 106. 
The appropriate advertisement on any given set top device 
would be the one where the local viewer usage profile 
matches or falls within the parameters of the required viewer 
usage profile attached to the advertisement. A default adver- 
tisement may be shown to homes where the viewer profile 
does not match or fall within the profiles or profile ranges 
requested by the advertisers targeting their advertisements to 
specific audiences. The required viewer usage profiles 
requested by the advertiser, and attached in some form to 
particular advertisements and insertions, may take the form 
of ranges of viewing intensity of one or more program 
categories or groups of categories, or it may be sent as a 
required range of ratios of the viewing intensity of one 
program category with respect to one or more other catego- 
ries. 

For instance, an advertiser may want their advertisements 
for a particular baby product shown only to house -holds 
which have a certain average viewing of both day time soap 
and day time children's shows but no significant viewing of 
sports. The reason for doing this may be because they 
believe that those parameters are an accurate way to target 
the families that are most likely to buy or use their products 
or services which may be, for instance, single parent moth- 
ers who have children and use live-in day care. Naturally, the 
same surveys used to compile the cross correlation keys 
would also be useful in estimating, to the same degree of 
accuracy, the size of the viewing population having viewer 
usage profiles that fit the advertisers requirements, and 
hence how much the advertisers should pay for their adver- 
tisements or insertions. 

An alternate way of expressing the required viewer usage 
profile may be to show the particular advertisement or 
insertion only to households where the ratios of the viewing 
intensity of some particular program categories, or groups of 
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categories, exceed some minimum, fall beneath some maxi- 
mum or he within some range of values. For instance, the 
requirement may be to only show the advertisements or 
insertions to the households where the ratio of average time 

5 spent viewing major league baseball to the time spent 
viewing situation comedies was one or greater. Or, the 
advertisers requirement may be that only households where 
the time spent watching day-time game shows was roughly 
equal to the time spent watching home shopping programs. 

3 0 The reason for requesting restriction to those house-holds 
would be because in the survey sample population these 
ratios had been found to be reliable indicators of some 
demographic or psychographic attribute of the viewer or 
viewing family that defined the advertiser's or other system 

25 user's desired audience. Program producers or broadcasters 
may themselves wish to use the system of Anonymous 
Profile targeting to present different versions of programs to 
different households. For instance, house-holds whose 
viewer usage profile indicated the presence of young chil- 

20 dren may be shown alternative versions of certain program 
in which either violent or sexually explicit scenes were 
either omitted, or had less violent or explicit scenes substi- 
tuted. It may be that at different times of the year the viewer 
usage profiles required to reach particular demographic or 

25 psychographic attributes change. 

Another function of central controller 146 may be to make 
the set-top device act as a downstream or "slave" section of 
an LVIS system. In particular central controller 146 would 
use the set-top device's 44 resources to strip off, interpret, 

30 and use the information attached to, or encoded in, the video 
or television signal by some up stream LVIS system 16. In 
particular, interpreter 164 would obtain information put in 
the video, television or data stream by recognition unit 18, 
tracking unit 20, occlusion mask production unit 22, and 

35 camera data interpreter 15 of the front-end or upstream LVIS 
system 16. Tne same sort of information provided by the 
front-end or upstream LVIS system 16 may also have been 
put in the video, television, or data stream by computer, as 
for instance, but not limited to, part of some video game, 

40 particularly multi-user video applications, or as part of a 
well known virtual studio set up. The LVIS information 
extracted by interpreter 164, may be temporarily stored in e 
data store 168 for use at a later, appropriate time, or used 
immediately to extract appropriate material from data store 

45 168, the video and audio storage unit 152, which may be 
video and audio insertions, and direct it via the appropriate 
additional desk top functional units. 

In addition, central controller 146 will use information 
about the required viewer usage profile attached to each 

50 proposed insert, and have compared it with viewer usage 
profile 120, stored in viewer usage profile store 170 to 
decide which insertion to use. The selected insertion is, if 
necessary, decompressed using video and audio decompres- 
sion unit 154, before being warped to the appropriate pose 

55 by video warping unit 160. Warper 160 is fed appropriate 
parameters via central controller 146, which has obtained 
them via interpreter 164. After warping, the insertion is 
mixed into the video and audio stream being sent to the end 
user's set 106 via video and audio mixer 156. This mixing 

60 also includes an occlusion mask, which has been generated 
by occlusion mask generator 158, which has also received 
parameters supplied upstream via video and audio inter- 
preter 164, and has also been warped into the correct pose 
via warper 160 using parameters supplied upstream by 

65 interpreter 164. The result is that the viewer sees an insertion 
on the viewing screen which appears to be part of the 
original video television, data broadcast, or transmission, 
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but which has been put there by the combined action of the 
original upstream LVIS, or other placing mechanism, and 
the set-top device, and is further dependent on the local 
viewer usage profile or history of the viewer's usage of that 
particular set. 

In a further embodiment, viewer usage profile 120 stored 
in viewer usage profile store 170 may be related to a viewer 
access key, or other form of identification, so that the viewer 
usage profile relates to a specific individual. 

In a still further embodiment of the invention, viewer 
usage profile store 170 may be totally; or in part, located 
inside the viewer's remote control 71, or other related device 
that the viewer uses to access and control the content 
reaching the end user display or television set. This related 
control device, includes, but is not limited to, hand held 
computers, personal computers, joy sticks, web browsers, 
and other similar hardware or software modules that may be 
used to control the data. In addition, the view or usage 
profile 120 stored in the viewer usage profile is linked by 
access number, or other suitable identification means, to 
either an individual person or individual device, or module, 
so that profiles may be constructed and stored for those 
individuals or individual device or software module. In 
addition, an individual device or software module may 
construct and store viewer usage profiles 120 for a number 
of different individuals, who may be identified by name, 
password, number, or other suitable identification means, 
including but not limited to biometric means such as 
signature, fingerprint, or retina pattern. 

It is to be understood that the apparatus and method of 
operation taught herein are illustrative of the invention. 
Modifications may readily be devised by those skilled in the 
art without departing from the spirit or scope of the inven- 
tion. 

What is claimed is: 

1. A set- top device for sending and receiving data per- 
taining to television or video viewing in which a video signal 
having been transmitted by a video distribution mechanism 
is received by said device and modified prior to viewing by 
utilizing automatically selected video indicia or sequences 
which are stored locally on said set-top device, said device 
comprising: 

means local to the user for monitoring the usage of a 
television or a video viewing device; 

means local to the user for automatically creating a 
continuously updated version of a viewer profile based 
upon the cumulative data acquired by said momtoring 
means, wherein said cumulative data acquired by said 
monitoring means is weighted by how recently it was 
collected; 

means local and private to the user for storing said viewer 
profile in said device; 

means remote from the user for transmitting demographic 
requirements of a target viewer into corresponding 
ranges of viewer profiles: 

means remote from the user for creating a table associ- 
ating said ranges of viewer profiles with said video 
indicia or sequences; 

means remote from the user for encoding data pertaining 
to where and when to insert video indicia or sequences 
for transmission on said video distribution mechanism; 

means local to the user for receiving and decoding said 
video signal and receiving, decoding and storing said 
table; 

means local to the user for receiving and decoding said 
data pertaining to where and when to insert said video 
indicia or sequences; 
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means local to the user for comparing the locally stored 
viewer profile with the ranges of viewer profiles in said 
stored table to determine the corresponding video indi- 
cia or sequences; and, 
means for inserting said video indicia or sequences 
directly into said video signal for viewing on said 
television or video viewing device, said inserting 
means further comprising using said data to determine 
where and when to insert said video indicia or 
sequences into said video signal. 
2. The set-top device of claim 1 further comprising: 
means for sampling a statistically representative sub-set 
of viewers to create statistical correlations between 
viewer demographics and said ranges of viewer pro- 
files. 

15 3. The set-top device of claim 2 wherein said monitoring 
means monitors time of day, day of week, duration of 
viewing time per channel, channel tuned, program type 
being viewed, and channel surfing patterns. 

4. The set-top device of claim 3 wherein said viewer 
20 profile includes information about the viewer derived from 

the monitoring means, said information designed to predict 
the age, sex, family size, hobbies and interests of the viewer. 

5. The set-top device of claim 4 wherein the statistical 
correlation between viewer demographics and said ranges of 

25 viewer profiles are used so that specific video sequences or 
indicia are anonymously targeted to specific viewer demo- 
graphics. 

6. The set-top device of claim 5 wherein the insertable 
indicia and the data pertaining to said insertable indicia are 
encoded into the vertical blanking interval, a spare audio 
channel, in the video itself, or in some other spare channel 
or field. 

7. The set-top device of claim 5 wherein a viewer can 
interactively amend, change, add to, or alter the viewer 
profile primarily for the purpose of soliciting specific indi- 
cia. 

8. The set- top device of claim 5 wherein the video signal 
can be a compressed video signal that is decompressed by 
said set-top device, or an analog or digital television broad- 
cast. 

9. The set-top device of claim 5 wherein said means for 
receiving includes a coaxial cable connection, television 
aerial connection, satellite broadcast connection, or tele- 
phone connection. 

10. The set-top device of claim 5 wherein the television 
can be substituted with a computer or any other video 
reception device. 

11. The set-top device of claim 10 wherein said indicia 
include links to Internet web-sites. 

12. A set-top device for sending and receiving data 
pertaining to television or video viewing in which a video 
signal having been transmitted by a video distribution 
mechanism is received by said device and modified prior to 
viewing by utilizing automatically selected video indicia or 
sequences which are stored locally on said set- top device, 
said device comprising: 

means local to the user for automatically creating a 
continuously updated viewer profile based upon the 
cumulative data acquired by said monitoring means, 
and wherein said cumulative data acquired by said 
monitoring means is weighed by how recently it was 
collected; 

means local and private to the user for storing said viewer 

profile in said device; 
means remote from the user for translating demographic 
requirements of a targeted viewer into corresponding 
ranges of viewer profiles; 
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means remote form the user for creating a table associ- 
ating said ranges of viewer profiles with said video 
indicia or sequences; 

means remote from the user for encoding said table for 
transmission on said video distribution mechanism; 

means local to the user for receiving and decoding said 
video signal and receiving, decoding and storing said 
table; 

means local to the user for comparing the locally stored 
viewer profile with the ranges of viewer profiles in said 
stored table to determine the corresponding video indi- 
cia or sequences; and, 

means for inserting said video indicia or sequences 
directly into said video signal for viewing on said 
television or video viewing device, and wherein said 
inserted indicia are interactive links to a web address, 
the contents of which may be accessed via the video 
distribution media or an associated data network and 
displayed on said television or video viewing device as 
part of said modified video signal or as a hypertext 
page. 

13. A set-top device for sending and receiving data 
pertaining to television or video viewing in which a video 
signal having been transmitted by a video distribution 
mechanism is received by said device and modified prior to 
viewing by utilizing automatically selected video indicia or 
sequences which are stored locally on said set-top device, 
said device comprising: 

means local to the user for monitoring the usage of a 
television or video viewing device; 

means local and private to the user for automatically 
creating a continuously updated viewer profile based 
upon the cumulative data acquired by said monitoring 
means, wherein said means for automatically creating 
a viewer profile further comprises analysis of the user's 
accesses to web-sites when browsing the World Wide 
Web or other computer network. 

14. A set-top device for sending and receiving data 
pertaining to television or video viewing in which a video 
signal having been transmitted by a video distribution 
mechanism is received by said device and modified prior to 
viewing by utilizing automatically selected video indicia or 
sequences which are stored locally on said-set top device, 
said device comprising: 

means local to the user for monitoring the usage of a 
television or video viewing device; 

means local to the user for automatically creating a 
continuously updated viewer profile based upon the 
cumulative data acquired by said monitoring means, 
wherein said cumulative data acquired by said moni- 
toring means is weighed by how recently it was col- 
lected; 

means local and private to the user for storing said viewer 

profile in said device; 
means remote from the user for translating demographic 

requirements of a targeted viewer into corresponding 

ranges of viewer profiles; 
means remote from the user for encoding said table for 

transmission on said video distribution mechanism; 
means local to the user for receiving and decoding said 

video signal and receiving, decoding and storing said 

table; 

means local to the user for comparing the locally stored 
viewer profile with the ranges of viewer profiles in said 
stored table to determine the corresponding video indi- 
cia or sequences; and, 

means for inserting said video indicia or sequences 
directly into said video signal for viewing on said 
television or video viewing device. 
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15. A set-top device for sending and receiving data 
pertaining to television or video viewing in which a video 
signal having been transmitted by a video distribution 
mechanism is received by said device and modified prior to 
viewing by utilizing automatically selected video indicia or 
sequences which are stored locally in said set top device, 
said device comprising: 

means local and private to the user for monitoring the 
usage of a television or video viewing device; 

means local to the user for automatically creating a 
continuously updated viewer profile based upon the 
cumulative data acquired by said monitoring means, 
wherein said viewer profile includes information about 
the viewer derived from the monitoring means, said 
information designed to predict the age, sex, family 
size, hobbies and interests of the view, and wherein said 
cumulative data acquired by said monitoring means is 
weighed by how recently it was collected; 

means local to the user for storing said viewer profile in 
said device; 

means remote from the user for translating demographic 
requirements of a targeted viewer into corresponding 
ranges of viewer profiles; 

means remote from the user for creating a table associ- 
ating said ranges of viewer profiles with said video 
indicia or sequences; 

means remote from the user for encoding said table for 
transmission on said video distribution mechanism; 

means local to the user for receiving and decoding said 
video signal and receiving, decoding and storing said 
table; 

means local to the user for comparing the locally stored 
viewer profile with the ranges of viewer profiles in said 
stored table to determine the corresponding video indi- 
cia or sequences; and, 

means for inserting said video indicia or sequences 
directly into said video signal for viewing on said 
television or video viewing device. 

16. A method of anonymous targeted insertion of selected 
indicia or sequences, which are stored locally on a set-top 
device, into a video broadcast comprising the steps of: 

a. monitoring the usage and viewing habits of viewer of 
a television set or other video reception device via said 
set-top device located at the television set or other 
video reception device, said monitoring step including 
the step of acquiring cumulative data weighted by how 
recently it was collected; 

b. automatically creating a continuously updated local 
viewer profile in said device derived from data acquired 
from said monitoring step, said viewer profile indicat- 
ing certain characteristics of the viewer; 

c. linking specific insertable indicia with specific viewer 
profiles; 

d. encoding insertable indicia as well as data pertaining to 
the placement, shape, size, and perspective of the 
indicia directly into the video broadcast; 

e. employing said locally and privately stored viewer 
profile to decide which insertable video indicia to insert 
based upon viewer profiles encoded in said video 
broadcast; and, 

f. having said set- top device decode the broadcast video 
and perform the insertion of the indicia. 
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